Assigment Week 6 Deep Learning Exercise. Image classification.

The goal of this assignment is to test and compare some simple deep learning architectures for the problem of 
image classification. We will be using the Tensorflow framework. It can be installed as explained at the end of this 
document. 
The CIFAR-10 dataset contains 60 000 images divided in 10 classes. The set is split in 50000 and 10000 samples 
in the training and testing set, respectively. The tutorial found at [https://www.tensorflow.org/tutorials/images/cnn] 
gives a good introduction to this practical exercise. The essential code in this tutorial can be found in the provided 
script 'ConvNet.py' or as a Jupyter notebook in [https://storage.googleapis.com/tensorflow_docs/docs/site/en/tutorials/images/cnn.ipynb]

EXERCISES

1- Multi Layer Perceptron (MLP). Modify the provided script 'perceptron.py' to build a MLP. Use architectures 
with 0, 1 and 2 hidden layers. Keep the complexity of the model bounded so runs do not take much more
than 1 hour to reach the maximum of testing accuracy. Notice that the input needs to be "flattened" since there is no spatial structure 
in this fully connected design.  This can be achieved by adding a dummy layer with no free parameters with "layers.Flatten()"
as the first layer in the constructor "model.Sequential()". Obtain the learning curves and discuss the results.
Report the optimizer in use, initialization parameters, the learning rate, etc. Is early stopping convenient
in this model?

2- Reuse the code from part 1 to build and run a MLP with one hidden layer as big a you can. 
Compare the performance of your design with the results appearing in Table 1 of [https://arxiv.org/pdf/1611.03530.pdf] for a MLP of 512 units in a single 
hidden layer. Report the best result found for a maximum of 1000 epochs or 2 hrs CPU running time.
The best accuracy amongst all teams will be awarded extra points.

3- Convolutional Networks. Study the performance properties of the convolutional network provided in the Tensorflow tutorial. How is 
the learning affected if instead of ReLU units, tanh() activations are used? What is the reason for this? Compare also
at least two different optimizer algorithms.

4- Try to outperform the convolutional network of part 3 with a MLP that uses approximately the same number of parameters.
Report your results and explain them.


Tensorflow INSTALLATION GUIDE:

To install Tensorflow, we advise you to use the conda package manager,
obtainable from [https://www.anaconda.com/distribution]. You probably want the
python 3.7 version. 

Install per the directions on the website, 

Make a separate environment (called ML_week6 in this example) for this
assignment, by executing
   $ conda create -n'ML_week6' python=3

Activate this environment using
   $ conda activate ML_week6
Note: in older version of conda this was done with
   $ source activate ML_week6

and install Tensorflow version 1.14.0 (version 2.0 has just been released, and
things work a little differently in that version) and matplotlib using 
   $ conda install tensorflow=1.14.0 matplotlib
 
Check if you have installed the right packages by opening a python shell and
executing `import tensorflow as tf`, then try whether the provided
`perceptron.py` script works.
Make sure you have the ML_week6 environment enabled when you open the python
shell (the activate command above). 

Note: If you have a CUDA-capable GPU and want to use that, you can follow the
guide on the  Tensorflow website [https://www.tensorflow.org/install/gpu]. But
we won't be able to help with installing it this way.

